The present embodiments relate to tracking anatomical structures in images of a patient. During surgery, surgeons often use fluoroscopy as a visual aid to perform the procedure. However, the position of anatomical structures may not be clearly visible in fluoroscopic images and may be difficult to track visually due to cardiac, as well as respiratory, movement. Existing methods require an operator to carefully mark a predefined set of landmarks for the tracking of an anatomical structure. Often these markers must be updated during the procedure. Although detection based trackers that employ offline trained detectors perform well for tracking known structures, their performance to track certain anatomical structures without a known shape in advance is still limited. One method currently in use to track the position and orientation of, for example, an aortic root is by injecting an iodinated contrast agent (angiography) into the patient. However, such contrast agents are harmful for the patient and their use should be minimized. Due to this, surgeons often must carefully weigh the benefits of the use of contrast agents for a successful procedure compared to the patient's safety due to the contrast agent's side effects to the patients. Alternatively, semi-supervised trackers that learn an objects appearance perform well but are very sensitive to initialization.